Method for multi-sensor white balance synchronization and electronic device using the same

ABSTRACT

A method of multi-sensor white balance synchronization, and an electronic device using the same, including sensing a same scene by a plurality of sensors of an imaging system to obtain and provide outputs of the plurality of sensors; obtaining color information of the plurality of sensors and color statistical information of the plurality of sensors from the outputs of the plurality of sensors; merging the color information of the plurality of sensors based on the color statistical information of the plurality of sensors, to obtain global color information; calculating white balance gain coefficients of each of the plurality of sensors using the global color information; and generating an image having adjusted white balance based on the outputs of the plurality of sensors and the white balance gain coefficients.

BACKGROUND

The present disclosure relates to automatic white balance (AWB)synchronization, and more particularly to methods of automatic whitebalance synchronization of an imaging system including a plurality ofsensors, and/or electronic devices using the same.

In an imaging system having a plurality of sensors, it is necessary toconsider the white balance difference among the plurality of sensors. Acurrent method of multi-sensor white balance synchronization includesselecting a sensor (e.g., a main camera) as a reference sensor, andperforming a white balance algorithm based on an image captured by thereference sensor, to obtain color information under a current capturingcondition (e.g., color temperature). Then, white balance gaincoefficients of the plurality of sensors are obtained according tocalibration characteristics between the reference sensor and othersensors based on the color information, so as to ensure that whitebalance effects of different sensors are consistent.

However, the color information obtained from the reference sensor maynot be accurate. In some cases, not enough information for automaticwhite balance calculation may be obtained by only using the referencesensor. There is thus a need for improved multi-sensor white balancesynchronization technology.

SUMMARY

Embodiments of the inventive concepts provide a method for multi-sensorwhite balance synchronization including sensing a same scene by aplurality of sensors of an imaging system to obtain and provide outputsof the plurality of sensors; obtaining, by a processor, colorinformation of the plurality of sensors and color statisticalinformation of the plurality of sensors from the outputs of theplurality of sensors; merging, by the processor, the color informationof the plurality of sensors based on the color statistical informationof the plurality of sensors, to obtain global color information;calculating, by the processor, white balance gain coefficients of eachof the plurality of sensors using the global color information; andgenerating, by the processor, an image having adjusted white balancebased on the outputs of the plurality of sensors and the white balancegain coefficients.

Embodiments of the inventive concepts further provide an electronicdevice including a plurality of sensors that sense a same scene toobtain and provide a plurality of sensor outputs; and a processor thatobtains color information of the plurality of sensors and colorstatistical information of the plurality of sensors based on theplurality of sensor outputs, merges the color information of theplurality of sensors based on the color statistical information of theplurality of sensors to obtain global color information, determineswhite balance gain coefficients of each of the plurality of sensorsusing the global color information, and generates an image havingadjusted white balance based on the plurality of sensor outputs and thewhite balance gain coefficients.

Embodiments of the inventive concepts still further provide anon-transitory computer-readable storage medium for storing instructionsexecutable by a processor for controlling an electronic device includinga plurality of sensors and the processor, the plurality of sensors sensea same scene to obtain and provide a plurality of sensor outputs, thenon-transitory computer-readable storage medium including a firstinstructionfor obtaining color information of the plurality of sensorsand color statistical information of the plurality of sensors based onthe plurality of sensor outputs; a second instruction for merging thecolor information of the plurality of sensors based on the colorstatistical information of the plurality of sensors to obtain globalcolor information; a third instruction for determining white balancegain coefficients of each of the plurality of sensors using the globalcolor information; and a fourth instruction for generating an imagehaving adjusted white balance based on the plurality of sensors and thewhite balance gain coefficients.

The method for multi-sensor white balance synchronization and theelectronic device according to some example embodiments of inventiveconcepts may have an effect of more accurate white balancesynchronization.

BRIEF DESCRIPTION OF THE DRAWINGS

The above and other objects, features and advantages will be moreclearly understood from the following detailed description together withthe accompanying drawings in which:

FIG. 1 illustrates a block diagram of an electronic device according toembodiments of the inventive concepts;

FIG. 2 illustrates a flowchart descriptive of a method of multi-sensorwhite balance synchronization according to embodiments of the inventiveconcepts;

FIG. 3 illustrates a flowchart descriptive of a method of multi-sensorwhite balance synchronization according to embodiments of the inventiveconcepts; and

FIG. 4 illustrates a block diagram of a mobile terminal according toembodiments of the inventive concepts.

DETAILED DESCRIPTION

As is traditional in the field of the inventive concepts, embodimentsmay be described and illustrated in terms of blocks which carry out adescribed function or functions. These blocks, which may be referred toherein as units or modules or the like, are physically implemented byanalog and/or digital circuits such as logic gates, integrated circuits,microprocessors, microcontrollers, memory circuits, passive electroniccomponents, active electronic components, optical components, hardwiredcircuits and the like, and may optionally be driven by firmware and/orsoftware. The circuits may, for example, be embodied in one or moresemiconductor chips, or on substrate supports such as printed circuitboards and the like. The circuits constituting a block may beimplemented by dedicated hardware, or by a processor (e.g., one or moreprogrammed microprocessors and associated circuitry), or by acombination of dedicated hardware to perform some functions of the blockand a processor to perform other functions of the block. Each block ofthe embodiments may be physically separated into two or more interactingand discrete blocks without departing from the scope of the inventiveconcepts. Likewise, the blocks of the embodiments may be physicallycombined into more complex blocks without departing from the scope ofthe inventive concepts.

FIG. 1 illustrates a block diagram of an electronic device 100 accordingto e embodiments of the inventive concepts.

The electronic device according to some embodiments may be or includefor example a camera, a smart cellphone, a tablet personal computer(PC), a personal digital assistant (PDA), a portable multimedia player(PMP), an augmented reality (AR) device, a virtual reality (VR) device,various wearable devices (e.g., a smart watch, smart glasses, a smartbracelet, etc.), and the like. However, embodiments of the inventiveconcepts are not limited to these electronic devices, and otherembodiments may include any electronic devices having an image capturefunction.

As shown in FIG. 1 , the electronic device 100 according to embodimentsof the inventive concepts includes at least a sensor unit (e.g.,circuit) 110 and a processor 120.

The sensor unit 110 may include a plurality of sensors, for examplesensor 1, sensor 2,..., and sensor M respectively denoted as sensors110-1, 110-2,..., and 110-M. M may be an integer greater than one. Insome embodiments, the plurality of sensors may include different typesof sensors. For example, the sensor 110-1 may be an image sensor forcapturing images, and the sensor 110-2 may be a color sensor.

The output of an image sensor may be an image. The plurality of sensorsmay also include one or more color sensors. The color sensors may detectand output one or more of correlated color temperature (CCT),brightness, illuminance, and spectral power distribution (SPD) accordingto its specific type. For example, when the color sensor is a colortemperature sensor, its output is the color temperature of theenvironment.

The processor 120 may process the output of the sensor unit 110, so asto perform image processing such as a white balance synchronizationoperation.

When a same scene is sensed by the plurality of sensors 110-1 to 110-Mof the sensor unit 110 of the electronic device 100, the processor 120may receive outputs of the plurality of sensors; obtain colorinformation of each sensor and color statistical information of eachsensor from the outputs of the plurality of sensors; merge the colorinformation of the plurality of sensors based on the color statisticalinformation of the plurality of sensors, to obtain global colorinformation; and calculate the white balance gain coefficient of each ofthe plurality of sensors using the global color information.

The processor 120 may be implemented as hardware such as for example ageneral-purpose processor, an application processor (AP), an integratedcircuit dedicated to image processing, a field programmable gate array,or a combination of hardware and software.

In some example embodiments, the electronic device 100 may also includea memory (not shown). The memory may store data and/or software forimplementing the method for multi-sensor white balance synchronizationaccording to some example embodiments. When the processor 120 executesthe software, the method for multi-sensor white balance synchronizationaccording to some example embodiments may be implemented. The memory maybe implemented as part of processor 120, or as separate from processor120 within electronic device 100.

The method for multi-sensor white balance synchronization according tosome example embodiments is hereinafter described in connection withFIG. 2 .

FIG. 2 illustrates a flowchart descriptive of a method of multi-sensorwhite balance synchronization according to embodiments of the inventiveconcepts. The description of the method of white balance synchronizationas follows is made with reference to the electronic device 100 of FIG. 1, but may be applied to electronic devices of various otherconfigurations. Although FIG. 2 illustrates various steps, an order ofthe steps is not necessarily limited to the order presented in FIG. 2 .

Referring to FIG. 2 , in step S210 a same scene is sensed by theplurality of sensors 110-1 to 110-M of the sensor unit 110. Theprocessor 120 receives outputs of the plurality of sensors of the sensorunit 110.

In step S220, the processor 120 obtains the color information of eachsensor and the color statistical information of each sensor from theoutputs of the plurality of sensors 110-1 to 110-M.

The color information may indicate the actual light environment of thecaptured scene, and may be any color-related information used insubsequent white balance calculation. In some example embodiments, thecolor information may for example be at least one of correlated colortemperature, brightness, illuminance, spectral power distribution, andthe like.

The types of color information respectively obtained from the outputs ofthe plurality of sensors are the same. For example, when correlatedcolor temperature is used for white balance calculation, the correlatedcolor temperature of each sensor is obtained from the output of eachsensor.

For image sensors, the color information may be calculated based on theoutput image. In some example embodiments, the color information may becalculated for example based on the pixel value of the output image,shooting setting parameters (for example, ISO, aperture, and/or shuttertime, etc.), etc. However, embodiments of the inventive concepts are notlimited thereto, and any other method of obtaining color informationfrom the output of an image sensor may also be used.

The output of the color sensor may be used as color information. In someexample embodiments, the color information may also be calculated basedon the output of the color sensor.

The color information may be a one-dimensional vector or amulti-dimensional vector, depending on the type of parameters requiredby the automatic white balance algorithm specifically used by theprocessor 120 of the electronic device. For example, when two differenttypes of color information (for example, correlated color temperatureand illuminance) are used for white balance calculation, the correlatedcolor temperature and illuminance are obtained respectively from theoutput of each sensor, as the color information vector.

The color statistical information may reflect the sensing accuracy orconfidence of the corresponding sensor. In some example embodiments, thecolor statistical information may be represented by variance. In thiscase, when the color information is a one-dimensional vector, the colorstatistical information may be the variance of the color information,and when the color information is a multi-dimensional vector, the colorstatistical information may be a covariance matrix. However, theinventive concepts are not limited thereto, and the color statisticalinformation may be other statistical parameters that may reflect thesensing accuracy or confidence of the corresponding sensor.

The calculation of color statistical information may be based ontraining of actual data or simulated data. For example, when the colorstatistical information is the variance, the color statisticalinformation may be calculated based on the image information of theimage sensor (for example, information entropy) or the outputinformation of the color sensor (for example, correlated colortemperature, brightness, spectral power distribution, and/orilluminance, etc.). The color statistical information may also bedetermined by the manufacturer of the sensor.

In some example embodiments, the processor 120 may obtain colorinformation and color statistical information of the image sensor, fromthe output of the image sensor, and may obtain corresponding colorinformation and color statistical information from the output of thecolor sensor.

In step S230, the processor 120 merges the color information of theplurality of sensors based on the color statistical information of theplurality of sensors, to obtain global color information. The globalcolor information may be considered as an optimal estimate of the colorinformation under the current light environment.

In some example embodiments, the processor 120 may perform a weightedsummation on the color information of the plurality of sensors based onthe color statistical information of the plurality of sensors, to obtainthe global color information.

In some example embodiments in which the color information is aone-dimensional vector, the global color information may be calculatedbased on equation (1) as follows:

$\begin{matrix}{x = {\sum\limits_{i = 1}^{\text{M}}\left( {x_{i} \times \omega_{i}} \right)}} & \text{­­­(1)}\end{matrix}$

In equation (1), x_(i) may represent a color information value of ani-th sensor among the plurality of sensors, ω_(i) may represent a weightfor the color information x_(i) of the i-th sensor, x may represent theglobal color information, and M may represent the number of the sensors.

In some example embodiments, the weight for the color information ofeach sensor may be inversely proportional to the color statisticalinformation of the corresponding color information, and directlyproportional to the merged color statistical information. The mergedcolor statistical information may represent the merged value of thecolor statistical information of the plurality of sensors.

In some example embodiments, the weight may be calculated based on theequation (2) as follows:

$\begin{matrix}{\omega_{i} = \frac{e}{e_{i}},} & \text{­­­(2)}\end{matrix}$

As an example, e_(i) may represent the color statistical information ofthe color information of the i-th sensor, and e may represent the mergedcolor statistical information.

However, the calculation of the weight is not limited to equation (2).In some example embodiments, the weight ω_(i) may be further adjustedaccording to the actual white balance effect.

For example, in the actual white balance effect calibration, some scenesmay be selected for debugging. The scenes selected for debugging mayhave corresponding target white balance gain coefficient. When there isa large difference between the white balance gain coefficient obtainedaccording to equations (1) and (2) and the target white balance gaincoefficient, the weight ω_(i) can be adjusted, so that the white balancegain coefficient calculated according to the adjusted weight ω_(i)′ iscloser to the target white balance gain coefficient. In an example, theadjusted weight

${\text{ω}^{\prime}}_{i} = \frac{k_{i}\text{ω}_{i}}{\sum{k_{i}\text{ω}_{i}}},$

where the adjustment coefficient k_(i) may be a fixed coefficient. Forexample, k_(i) may be calculated by Try Out method or other methods,such as Newton down-hill method, such that the difference between thecalculated white balance gain coefficient and the target white balancegain coefficient is minimized. However, the above description is onlyexemplary, and the present invention is not limited thereto.

In some example embodiments, the merged color statistical information emay be calculated based on the equation (3) as follows:

$\begin{matrix}{\frac{1}{e} = {\sum\limits_{i = 1}^{\text{M}}\frac{1}{e_{i}}},} & \text{­­­(3)}\end{matrix}$

The merged value of the color statistical information e may becharacterized as a reciprocal of the sum of the reciprocals of the colorstatistical information e_(i) of the plurality of sensors.

However, the calculation of the merged color statistical information isnot limited to equation (3), and the color statistical information mayalso be merged by other merging algorithms.

The calculation method of the global color information in some exampleembodiments in which the color information is an n-dimensional vector (nis an integer greater than 1) is hereinafter described.

Assuming that the sensor unit 110 of the electronic device 100 includesM sensors, and n types of color information are obtained from the outputof each sensor 110-1 to 110-M, the color information of an i-th sensormay be expressed as a n-dimensional color information vector as shown inequation (4):

$\begin{matrix}{X_{i} = \left( {x_{i1}, \cdot x_{i2}, \cdot \cdots, \cdot x_{in}} \right)^{\text{T}}.} & \text{­­­(4)}\end{matrix}$

In equation (4), the superscript “T” indicates transpose of matrix. Inthe case where the color information is a multi-dimensional vector, thecolor statistical information of each sensor may be a color statisticalinformation matrix. The color statistical information matrix representsa predetermined statistical value among different color informationvalues of each sensor. For example, the color statistical informationmatrix E_(i) corresponding to the n-dimensional color information vectorof the i-th sensor may be expressed as:

$\begin{matrix}{E_{i} = \begin{bmatrix}e_{11i} & \cdots & e_{1ni} \\ \vdots & \ddots & \vdots \\e_{n1i} & \cdots & e_{nni}\end{bmatrix},} & \text{­­­(5)}\end{matrix}$

In equation (5), e_(pqi) is the predetermined statistical value for thep-th color information value and the q-th color information value of thei-th color sensor.

In some example embodiments, the predetermined statistical value may bea covariance, and the color statistical information matrix may be acovariance matrix. For example, in the case where the color statisticalinformation matrix is a covariance matrix, e_(11i) is the variance ofthe first color information value of the i-th color sensor, and e_(1ni)is covariance of the first color information value and the n-th colorinformation value of the i-th color sensor.

In some example embodiments, in the case where the color information isan n-dimensional vector, the color information of different sensors maybe weighted and merged using the weight corresponding the colorinformation of each sensor based on the following equation (6), so as toobtain the global color information:

$\begin{matrix}{X = {\sum\limits_{i = 1}^{\text{M}}{W_{i}X_{i}}}} & \text{­­­(6)}\end{matrix}$

Here, X_(i) may represent the color information vector of the i-thsensor among the plurality of sensors, W_(i) may represent the weightmatrix corresponding to the color information vector X_(i) of the i-thsensor, and X may represent the global color information vector.

In some example embodiments, the weight for the color information ofeach sensor may be inversely proportional to the color statisticalinformation of the corresponding color information (for example, thecolor statistical information matrix), and directly proportional to themerged color statistical information (for example, a merged colorstatistical information matrix). Specifically, the weight in the weightmatrix for the color information of each sensor may be inverselyproportional to the corresponding element in the corresponding colorstatistical information matrix, and directly proportional to thecorresponding element in the merged color statistical informationmatrix.

In some example embodiments, the weight matrix W_(i) may be calculatedbased on the color statistical information matrix of the respectivesensor, as shown in following equation (7):

$\begin{matrix}{W_{i} = E_{i}^{- 1}E.} & \text{­­­(7)}\end{matrix}$

In equation (7), E may represent the merged color statisticalinformation matrix.

However, the calculation of the weight matrix is not limited to equation(7). In some example embodiments, the weight matrix W_(i) may be furtheradjusted according to the actual white balance effect.

For example, in the actual white balance effect calibration, some scenesmay be selected for debugging. The scenes selected for debugging mayhave corresponding target white balance gain coefficient. When there isa large difference between the white balance gain coefficient obtainedaccording to equations (6) and (7) and the target white balance gaincoefficient, the weight matrix W_(i) may be adjusted, so that the whitebalance gain coefficient calculated according to the adjusted weightmatrix W_(i)′ is closer to the target white balance gain coefficient.However, the above description is only exemplary, and the presentinvention is not limited thereto.

In some example embodiments, the merged color statistical informationmatrix E may be calculated according to the equation (8) as follows:

$\begin{matrix}{E = \left( {\sum\limits_{\text{j} = 1}^{\text{M}}E_{\text{j}}^{- 1}} \right)^{- 1}} & \text{­­­(8)}\end{matrix}$

The merged color statistical information E may be characterized as aninverse of a sum of inverses of color statistical information matricesof a color information vector.

However, the calculation of the merged color statistical informationmatrix is not limited to equation (8), and the color statisticalinformation matrix may also be merged by other merging algorithms.

In step S240, the processor 120 calculates the white balance gaincoefficient of each of the plurality of sensors using the global colorinformation. Various general white balance algorithms in the prior artmay be used, so as to obtain the white balance gain coefficient usingthe obtained global color information.

In an example embodiment, a look-up table may be used to calculate thewhite balance gain coefficient with the obtained global colorinformation. For example, assuming that the obtained global colorinformation is correlated color temperature, the mapping relationshipbetween multiple correlated color temperature intervals and whitebalance gain coefficients may be established in advance in the form of alook-up table. The white balance gain coefficient corresponding to theglobal color information may be obtained using the lookup table afterthe global color information (i.e., correlated color temperature) isobtained.

In some example embodiments, the white balance gain coefficients may becalculated using various automatic white balance algorithms (such asautomatic white balance algorithms based on gray world, perfectreflection, dynamic threshold, etc., automatic white balance algorithmsbased on color temperature and illuminance, and the like) based onglobal color information.

The following expressions (9) and (10) show examples of calculatingwhite balance gain coefficients based on the global color information:

$\begin{matrix}\begin{array}{l}\left. CCT_{test}\rightarrow\left( {\frac{{\overline{r}}_{t}}{{\overline{g}}_{t}},\frac{{\overline{b}}_{t}}{{\overline{g}}_{t}}} \right) \right. \\\left. CCT_{ref}\rightarrow\left( {\frac{{\overline{r}}_{r}}{{\overline{g}}_{r}},\frac{{\overline{b}}_{r}}{{\overline{g}}_{r}}} \right) \right.\end{array} & \text{­­­(9)}\end{matrix}$

and

$\begin{matrix}{K = \alpha\left( {\frac{{\overline{r}}_{t}}{{\overline{g}}_{t}},\frac{{\overline{b}}_{t}}{{\overline{g}}_{t}}} \right) + \left( {1 - \alpha} \right)\left( {\frac{{\overline{r}}_{r}}{{\overline{g}}_{r}},\frac{{\overline{b}}_{r}}{{\overline{g}}_{r}}} \right)} & \text{­­­(10)}\end{matrix}$

In this example, the global color information (for example, the globalcolor information X in equation (6) obtained based on equations (7) and(8)) required to calculate the white balance gain coefficient includescorrelated color temperature and illuminance, wherein the correlatedcolor temperature in the global color information is reflected inexpression (9) (for example, CCT_(test)), and the illuminance in theglobal color information is reflected in expression (10) (for example,the illuminance may determine the value of the visual adaptation factorα).

Expression (9) represents mapping from the global color informationunder the shooting environment (i.e., correlated color temperatureCCT_(test)) to the gain coefficient, and mapping from the colorinformation under the reference environment (i.e., the correlated colortemperature CCT_(ref), for example, 6500 K) to the gain coefficient, andr_(t) , b_(t) , g_(t) , r_(r) , b_(r) and g_(r) represent the gaincoefficients of the three channels R, B, and G under the shootingenvironment and the reference environment, respectively. For example,based on expression (9), r_(t) , b_(t) and g_(t) may be obtained fromthe correlated color temperature CCT_(test) in the obtained global colorinformation, and r_(r) , b_(r) , g_(r) and CCT_(ref) may bepredetermined. Furthermore, the visual adaptation factor α can beobtained based on the illuminance in the global color information. Basedon Expression (10), the white balance gain coefficient K can becalculated.

In order to make the automatic white balance synchronization algorithmmore accurate, stable and smooth, the color information obtained mayalso be time filtered.

However, the above description is only exemplary, and the presentinvention is not limited thereto.

FIG. 3 illustrates a flowchart descriptive of a method of multi-sensorwhite balance synchronization according to embodiments of the inventiveconcepts. The description of the method of multi-sensor white balancesynchronization as follows is made with reference to the electronicdevice 100 of FIG. 1 , but may be applied to electronic devices ofvarious other configurations.

Referring to FIG. 3 , in step S310, the plurality of sensors 110-1 to110-M of the sensor unit 110 sense a same scene a plurality of times toobtain multiple output frames, with each output frame including outputsof the plurality of sensors obtained during each sensing. The processor120 receives the plurality of output frames from the sensor unit 110.

In step S320, the processor 120 obtains the color information of theplurality of sensors 110-1 to 110-M and the color statisticalinformation of the plurality of sensors 110-1 to 110-M from the outputsof the plurality of sensors in each output frame, as color informationof each output frame and color statistical information of each outputframe, respectively.

In step S331, the processor 120 merges the color information of eachoutput frame, to obtain the global color information of each outputframe. The merging performed by the processor 120 for the colorinformation of each output frame may be the same as in step S220described above with reference to FIG. 2 and detailed descriptionrepetitive of step 220 will be omitted for the sake of brevity.

In step S332, the processor 120 performs time filtering on the globalcolor information of the plurality of output frames, to obtain globalcolor information to be used for calculating the white balance gaincoefficient. The global color information to be used for calculating thewhite balance gain coefficient may be determined as an output value ofthe time filtering of the last frame among the plurality of outputframes.

Various time filtering methods may be used. In some example embodiments,a first-order filtering of infinite impulse response filtering (IIR) maybe used. In this case, the output of the current frame is a weighted sumof the output of the previous frame and the global color information ofthe current frame.

The following equation (11) shows an example of IIR time filtering inthe case where the color information is a one-dimensional vector:

$\begin{matrix}\begin{matrix}{x_{F}^{k} = \omega^{k - 1}x_{F}^{k - 1} + \omega^{k}x^{k},} \\{e_{F}^{k} = \left( {{\eta^{k - 1}/e_{F}^{k - 1}} + {\eta^{k}/e^{k}}} \right)^{- 1}.}\end{matrix} & \text{­­­(11)}\end{matrix}$

In equation (11), the superscript k represents the sequence number ofthe frame, and the subscript “F” represents the time filtered signal(i.e., the time filtered global color information or the time filteredmerged color statistical information). For example, x^(k) and e^(k)represent the global color information of the k-th frame and the mergedcolor statistical information of the k-th frame, respectively,

x_(F)^(k)

represents the filtered global color information of the k-th frame, and

e_(F)^(k)

represents the filtered merged color statistical information of the k-thframe. ω^(k) and ω^(k-1) represent the filter weights of the kth frameand the (k-1)th frame, respectively, η is the fading factor, and η=0~1.

The following equation (12) shows an example of calculating filterweights ω^(k) and ω^(k-1) based on color statistical information:

$\begin{matrix}\begin{matrix}{\omega^{k - 1} = \frac{\eta^{k - 1}/e_{F}^{k - 1}}{{\eta^{k - 1}/e_{F}^{k - 1}} + {\eta^{k}/e^{k}}},} \\{\omega^{k} = \frac{\eta^{k}/e^{k}}{{\eta^{k - 1}/e_{F}^{k - 1}} + {\eta^{k}/e^{k}}}.}\end{matrix} & \text{­­­(12)}\end{matrix}$

However, the time filtering of step S332 for the case where the colorinformation is a one-dimensional vector is not limited to equations (11)to (12), and in other embodiments the time filtering may be performed byother time filtering algorithms.

The example of time filtering when the color information is ann-dimensional vector is described hereinafter.

Assuming that the M sensors 110-1 to 110-M of the sensor unit 110satisfy the Gaussian independent distribution, and the fading factor ηbelongs to an interval of [0, 1], the IIR time filtered colorinformation vector and merged color statistical information matrix maybe calculated according to equations (13) and (14) .

$\begin{matrix}\begin{matrix}{X_{F}^{k} = W^{k - 1}X_{F}^{k - 1} + W^{k}X^{k},} \\{E_{F}^{k} = \left( {\eta^{k - 1}\left( E_{F}^{k - 1} \right)^{- 1} + \eta^{k}\left( E^{k} \right)^{- 1}} \right)^{- 1}.}\end{matrix} & \text{­­­(13)}\end{matrix}$

and

$\begin{matrix}\begin{array}{l}{W^{k - 1} = \eta^{k - 1}\left( E_{F}^{k - 1} \right)^{- 1}\left\lbrack {\eta^{k - 1}\left( E_{F}^{k - 1} \right)^{- 1} + \eta^{k}\left( E^{k} \right)^{- 1}} \right\rbrack^{- 1}} \\\left. \cdots\cdots = \eta^{k - 1}E^{k}\left( {\eta^{k - 1}E^{k} + \eta^{k}E_{P}^{k - 1}} \right)^{- 1}↵ \right. \\\left. W^{k} = \eta^{k}\left( E^{k} \right)^{- 1}\left\lbrack {\eta^{k - 1}\left( E_{F}^{k - 1} \right)^{- 1} + \eta^{k}\left( E^{k} \right)^{- 1}} \right\rbrack^{- 1}↵ \right. \\\left. \cdots\cdots = \eta^{k}E_{F}^{k - 1}\left( {\eta^{k - 1}E^{k} + \eta^{k}E_{F}^{k - 1}} \right)^{- 1}↵ \right.\end{array} & \text{­­­(14)}\end{matrix}$

In equations (13) and (14), the superscript k represents the sequencenumber of the frame, and the subscript “F” represents the time filteredsignal (i.e., the time filtered color information vector or the timefiltered merged color statistical information matrix). For example,X^(k) and E^(k) represent the global color information vector and themerged color statistical information matrix of the k-th frame,respectively,

X_(F)^(k)

represents the filtered global color information vector of the k-thframe,

E_(F)^(k)

represents the filtered merged color statistical information matrix ofthe k-th frame, and W^(k) represents the filter weight matrix of the kthframe.

However, the time filtering of step S332 for the case where the colorinformation is an n-dimensional vector is not limited to equations (13)to (14), and in other embodiments the time filtering may be performed byother time-domain filtering algorithms.

In step S340, the processor 120 calculates the white balance gaincoefficient of each sensor of the plurality of sensors using the globalcolor information, which is the output value of the time filtering ofthe last frame of the plurality of output frames. Various general whitebalance algorithms in the prior art may be used, so as to obtain thewhite balance gain coefficient using the obtained global colorinformation. In an example, a look-up table may be used to calculate thewhite balance gain coefficient with the obtained global colorinformation. However, the above algorithms are exemplary, and thepresent invention is not limited thereto.

Time filtering combines the information of the plurality of frames, suchthat the automatic white balance synchronization algorithm is moreaccurate, stable and smooth.

FIG. 4 illustrates a block diagram of a mobile terminal according toembodiments of the inventive concepts.

As shown in FIG. 4 , the mobile terminal 400 according to some exampleembodiments includes a sensor unit (circuit) 410, a controller 420, acommunication circuit 430, an input circuit 440, a storage 450, and adisplay 460. The mobile terminal 400 may include additional circuitry.

The sensor unit 410 is connected to the controller 420. The sensor unit410 is used to sense the scene. The sensor unit 410 may for example beconfigured such as sensor unit 110 of FIG. 1 including a plurality ofsensors 110-1 to 110-M. The controller 420 processes the signals sensedby the sensor unit 410 (for example, using the method for multi-sensorwhite balance synchronization shown in FIG. 2 ). The controller 420 maydisplay the processed image on the display 460 and/or may store theimage in the storage 450.

For example, sensor unit 410 may sense the scene using a plurality ofsensors such as sensors 110-1 to 110-M in FIG. 1 to generate and outputsignals. Controller 420 may process the signals output by sensor unit410 to adjust white balance using the white balance gain coefficientsgenerated using the method for multi-sensor white balancesynchronization described with respect to FIG. 2 . Controller 420 maythen generate an image having improved white balance based on thesignals adjusted using the white balance gain coefficients. The imagemay be displayed on display 460 or stored in storage 450.

The communication circuit 430 may perform a communication operation forthe mobile terminal with another terminal or a communication network.The communication circuit 430 may establish a communication channel tothe another terminal or the communication network and/or may performcommunication associated with, for example, an image processing.

The input circuit 440 may receive various input information and variouscontrol signals, and transmit the input information and control signalsto the controller 420. The input unit 440 may be realized by variousinput devices such as keypads and/or keyboards, touch screens and/orstyluses, etc., but is not limited thereto.

The storage 450 may include volatile memory and/or nonvolatile memory.The storage 450 may store various data generated and used by the mobileterminal. For example, the storage 450 may store an operating system andapplications (e.g. applications associated with the method of inventiveconcepts) for controlling the operation of the mobile terminal. Thecontroller 420 may control the overall operation of the mobile terminaland may control part or all of the internal elements of the mobileterminal. The controller 420 may for example be implemented as ageneral-purpose processor, an application processor (AP), an applicationspecific integrated circuit, and/or a field programmable gate array,etc., but is not limited thereto.

The methods that perform the operations described in this applicationmay be performed by computing hardware, for example, by one or moreprocessors or computers, implemented as described above executinginstructions or software to perform the operations described in thisapplication that are performed by the methods. For example, a singleoperation or two or more operations may be performed by a singleprocessor, or two or more processors, or a processor and a controller.One or more operations may be performed by one or more processors, or aprocessor and a controller, and one or more other operations may beperformed by one or more other processors, or another processor andanother controller. One or more processors, or a processor and acontroller, may perform a single operation, or two or more operations.

Instructions or software to control a processor or computer to implementthe hardware components and perform the methods as described above maybe written as computer programs, code segments, instructions or anycombination thereof, for individually or collectively instructing orconfiguring the processor or computer to operate as a machine orspecial-purpose computer to perform the operations performed by thehardware components and the methods as described above. In one example,the instructions and/or software include machine code that is directlyexecuted by the processor or computer, such as machine code produced bya compiler. In another example, the instructions or software includehigher-level code that is executed by the processor or computer using aninterpreter. Persons and/or programmers of ordinary skill in the art mayreadily write the instructions and/or software based on the blockdiagrams and the flow charts illustrated in the drawings and thecorresponding descriptions in the specification, which disclosealgorithms for performing the operations performed by the hardwarecomponents and the methods as described above.

For example, embodiments of the inventive concepts may include anon-transitory computer-readable storage medium for storing instructionsexecutable by processor 120 of FIG. 1 for controlling electronic device100 including sensor unit 110 having the plurality of sensors 110-1 to110-M and the processor 100. The plurality of sensors 110-1 to 110-M maysense a same scene to obtain and provide a plurality of sensor outputs.The non-transitory computer-readable storage medium may include a firstcode segment stored for controlling (i.e., instructing) processor 120 toobtain color information of the plurality of sensors 110-1 to 110-M andcolor statistical information of the plurality of sensors 110-1 to 110-Mbased on the plurality of sensor output. The non-transitorycomputer-readable storage medium may include a second code segment forcontrolling (i.e., instructing) processor 120 to merge the colorinformation of the plurality of sensors 110-1 to 110-M based on thecolor statistical information of the plurality of sensors 110-1 to 110-Mto obtain global color information. The non-transitory computer-readablestorage medium may include a third code segment for controlling (i.e.,instructing) processor 120 to determine white balance gain coefficientsof each of the plurality of sensors 110-1 to 110-M using the globalcolor information. The non-transitory computer-readable storage mediummay include a fourth code segment for controlling (i.e., instructing)processor 120 to generate an image having adjusted white balance basedon the plurality of sensors and the white balance gain coefficients.

For example, embodiments of the inventive concepts may include anon-transitory computer-readable storage medium for storing instructionsexecutable by controller 420 of FIG. 4 for controlling mobile terminal(device) 400 similar to the described above with respect to thenon-transitory computer-readable storage medium described with respectto FIG. 1 . In other embodiments, the non-transitory computer-readablestorage medium may include additional code segments for controlling(i.e., instructing) processor 120/controller 420 to display the image ona display (e.g., display 460 of FIG. 4 ) and/or to store the image inmemory (e.g., storage 450 of FIG. 4 ).

The instructions or software to control a processor or computer toimplement the hardware components and perform the methods as describedabove, and any associated data, data files, and data structures, may berecorded, stored, or fixed in or on one or more non-transitorycomputer-readable storage media. Examples of a non-transitorycomputer-readable storage medium include at least one of read-onlymemory (ROM), random-access programmable read only memory (PROM),electrically erasable programmable read-only memory (EEPROM),random-access memory (RAM), dynamic random access memory (DRAM), staticrandom access memory (SRAM), flash memory, non-volatile memory, CD-ROMs,CD-Rs, CD+Rs, CD-RWs, CD+RWs, DVD-ROMs, DVD-Rs, DVD+Rs, DVD-RWs,DVD+RWs, DVD-RAMs, BD-ROMs, BD-Rs, BD-R LTHs, BD-REs, blue-ray oroptical disk storage, hard disk drive (HDD), solid state drive (SSD),flash memory, a card type memory such as multimedia card or a micro card(for example, secure digital (SD) or extreme digital (XD)), magnetictapes, floppy disks, magneto-optical data storage devices, optical datastorage devices, hard disks, solid-state disks, and any other devicethat is configured to store the instructions or software and anyassociated data, data files, and data structures in a non-transitorymanner and providing the instructions or software and any associateddata, data files, and data structures to a processor or computer so thatthe processor or computer may execute the instructions.

While various example embodiments have been described, it should beapparent to one of ordinary skill in the art that various changes inform and detail may be made in these examples without departing from thespirit and scope of the claims and their equivalents.

What is claimed is:
 1. A method of multi-sensor white balancesynchronization comprising: sensing a same scene by a plurality ofsensors of an imaging system to obtain and provide outputs of theplurality of sensors; obtaining, by a processor, color information ofthe plurality of sensors and color statistical information of theplurality of sensors from the outputs of the plurality of sensors;merging, by the processor, the color information of the plurality ofsensors based on the color statistical information of the plurality ofsensors, to obtain global color information; calculating, by theprocessor, white balance gain coefficients of each of the plurality ofsensors using the global color information; and generating, by theprocessor, an image having adjusted white balance based on the outputsof the plurality of sensors and the white balance gain coefficients. 2.The method of multi-sensor white balance synchronization of claim 1,wherein the merging the color information of the plurality of sensorsbased on the color statistical information of the plurality of sensorscomprises performing a weighted summation on the color information ofthe plurality of sensors based on the color statistical information ofthe plurality of sensors, wherein a weight for the color information ofeach sensor of the plurality of sensors is inversely proportional to thecolor statistical information of the color information, and is directlyproportional to merged color statistical information, and wherein themerged color statistical information represents a merged value of thecolor statistical information of the plurality of sensors.
 3. The methodof multi-sensor white balance synchronization of claim 2, wherein themerged value of the color statistical information of the plurality ofsensors is a reciprocal of a sum of reciprocals of the color statisticalinformation of the plurality of sensors, or an inverse of a sum ofinverses of color statistical information matrices of a colorinformation vector.
 4. The method of multi-sensor white balancesynchronization of claim 1, wherein the color statistical informationcomprises a variance of the color information.
 5. The method ofmulti-sensor white balance synchronization of claim 1, wherein thesensing the same scene by the plurality of sensors of the imaging systemcomprises sensing the same scene a plurality of times by the pluralityof sensors of the imaging system to obtain multiple output frames, eachoutput frame of the multiple output frames including outputs of theplurality of sensors obtained during each sensing, wherein the obtainingthe color information of the plurality of sensors and the colorstatistical information of the plurality of sensors from the outputs ofthe plurality of sensors comprises obtaining the color information ofthe plurality of sensors and the color statistical information of theplurality of sensors from the outputs of the plurality of sensors ineach output frame, as color information of each output frame and colorstatistical information of each output frame, respectively, wherein themerging the color information of the plurality of sensors based on thecolor statistical information to obtain the global color informationcomprises merging of the color information of each output frame toobtain global color information of each output frame, and performingtime filtering on the global color information of each output frame, toobtain global color information for calculating the white balance gaincoefficients, and wherein time filtered global color information of alast frame from among the multiple output frames is used as the globalcolor information for calculating the white balance gain coefficients.6. The method of multi-sensor white balance synchronization of claim 1,wherein the color information is color-related information for whitebalance calculation, and types of color information respectivelyobtained from the outputs of the plurality of sensors are the same. 7.The method of multi-sensor white balance synchronization of claim 1,wherein the color information comprises at least one of correlated colortemperature, brightness, illuminance, and spectral power distribution.8. The method of multi-sensor white balance synchronization of claim 1,wherein the plurality of sensors include a first type of sensor and asecond type of sensor different from the first type of sensor, whereinthe first type of sensor is an image sensor, and an output of the firsttype of sensor is an image, and wherein the second type of sensor is acolor sensor, and an output of the second type of sensor includes atleast one of correlated color temperature, brightness, illuminance, andspectral power distribution.
 9. An electronic device comprising: aplurality of sensors configured to sense a same scene to obtain andprovide a plurality of sensor outputs; and a processor configured toobtain color information of the plurality of sensors and colorstatistical information of the plurality of sensors based on theplurality of sensor outputs, merge the color information of theplurality of sensors based on the color statistical information of theplurality of sensors to obtain global color information, determine whitebalance gain coefficients of each of the plurality of sensors using theglobal color information, and generate an image having adjusted whitebalance based on the plurality of sensor outputs and the white balancegain coefficients.
 10. The electronic device of claim 9, wherein theprocessor is configured to merge the color information of the pluralityof sensors by performing a weighted summation on the color informationof the plurality of sensors based on the color statistical informationof the plurality of sensors, wherein a weight for the color informationof each sensor of the plurality of sensors is inversely proportional tothe color statistical information of the color information, and isdirectly proportional to merged color statistical information, andwherein the merged color statistical information represents a mergedvalue of the color statistical information of the plurality of sensors.11. The electronic device of claim 10, wherein the merged value of thecolor statistical information of the plurality of sensors is areciprocal of a sum of reciprocals of the color statistical informationof the plurality of sensors, or an inverse of a sum of inverses of colorstatistical information matrices of a color information vector.
 12. Theelectronic device of claim 9, wherein the color statistical informationcomprises a variance of the color information.
 13. The electronic deviceof claim 9, wherein the plurality of sensors are configured to sense thesame scene a plurality of times to obtain multiple output frames, eachoutput frame of the multiple output frames including outputs of theplurality of sensors obtained during each sensing, wherein the processoris configured to obtain the color information and the color statisticalinformation by obtaining the color information of the plurality ofsensors and the color statistical information of the plurality ofsensors from the plurality of sensor outputs of the plurality of sensorsin each output frame as color information of each output frame and colorstatistical information of each output frame, respectively, wherein theprocessor is configured to merge the color information by merging of thecolor information of each output frame to obtain global colorinformation of each output frame, and performing time filtering on theglobal color information of each output frame, to obtain global colorinformation for determining the white balance gain coefficient, andwherein time filtered global color information of a last frame fromamong the multiple output frames is used as the global color informationfor determining the white balance gain coefficients.
 14. The electronicdevice of claim 9, wherein the color information is color-relatedinformation for white balance determination, and types of colorinformation respectively obtained from the plurality of sensor outputsare the same.
 15. The electronic device of claim 9, wherein the colorinformation comprises at least one of correlated color temperature,brightness, illuminance, and spectral power distribution.
 16. Theelectronic device of claim 9, wherein the plurality of sensors include afirst type of sensor and a second type of sensor different from thefirst type of sensor, wherein the first type of sensor is an imagesensor, and a sensor output of the first type of sensor is an image, andwherein the second type of sensor is a color sensor, and a sensor outputof the second type of sensor includes at least one of correlated colortemperature, brightness, illuminance, and spectral power distribution.17. The electronic device of claim 9, further comprising a displayconfigured to display the image, the electronic device configured as amobile device.
 18. The electronic device of claim 9, further comprisinga memory configured to store the image, the electronic device configuredas a mobile device.
 19. A non-transitory computer-readable storagemedium for storing instructions executable by a processor forcontrolling an electronic device including a plurality of sensors andthe processor, the plurality of sensors configured to sense a same sceneto obtain and provide a plurality of sensor outputs, the non-transitorycomputer-readable storage medium comprising: a first instruction forobtaining color information of the plurality of sensors and colorstatistical information of the plurality of sensors based on theplurality of sensor outputs; a second instruction for merging the colorinformation of the plurality of sensors based on the color statisticalinformation of the plurality of sensors to obtain global colorinformation; a third instruction for determining white balance gaincoefficients of each of the plurality of sensors using the global colorinformation; and a fourth instruction for generating an image havingadjusted white balance based on the plurality of sensors and the whitebalance gain coefficients.